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ABSTRACT 

A new procedure is proposed for developing^ 

evaluating, and implementing routing procedures for use with 

individualized instruction programs^ An item response theory model 
for describing the relat ioniship Between performance oh sequentially 
arranged uhitis 6£ instruct ioh^ and procedures for using the model to 
evaluate sequential relationships and £6r making routing decisions. _ 
are discussed.. A procedure for estimating the parameters of the model 
is described. Empi r ical data are presented to support the validity of 
the model. The model and procedures described appear to be useful aifid 
to merit continued research efforts directed toward their 
development. (DWH) 
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One of the fastest growinjr areas in the field of education today is the 
area of individualized instruction - instruction in which content^ 
organization, or pacing is modified for each individual, Althou)?^h 
individualized instruction comes in many forms (e.^,, personalized systems of 
instruction, computer assisted instruction, individually prescribed 
instruction, and programmed instruction), all of these forms share the same 
baste design. All are basically sequences of instructional units through 
which subjects are routed by means of a series of tests. 

The way in which the units of instruction are sequenced arid the routing 
decisions made are two of the more crucical components of any iridividualized 
instruction program- While they have been the topic of corisiderable research, 
as yet no generally accepted procedures have beeri developed for these 
components. The purpose of this paper is to propose a new procedure for 

developing, evaluating^ and implementing rbutirig procedures for use with 

individualized instruction programs. Specifically, a model wili be proposed 
for describing the relationship between performance on sequentially arranged 
units of instruction, and procedures for using the model to evaluate 
sequential relatibriships arid for making routing decisions will be discussed. 
Then a procedure for estimating the parameters of the model will be 
discussed. Finally, empirical data wili be presented to support the validity 
of the model. Before beginning the discussion of this procedure, however, 
some theory abdut the nature of sequential units of instruction will be 
presented as a basis for the procedure. 

Sequential Units of Iris t ruction 

Underlying Theory 

The basic assumptiori uriderlying the sequential arrangement of anits of 
iristructibn is that perfbrmarice bn mbdule 2 requires the prior knowiedge of 
the material contained in mbdule I. It might be true that all material in 
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module 1 must be mastered before any (or at least any appreciable, amburit) of 
the material in modaie 2 can be mastered, or it may be . the case that certain 
sections of modnle t are prerequisit:e for certain sections of module 2. For 
this paper, the former will be assumed to be the case. It will also be 
assumed that the tests that measure the skills taiioiht in the modules are 
unidimensional • 

To say that a sequential relationship exists means that a certain level 
of performance is required on module 1 (mp before tearninc^ on module 2 (m^) 

can bee;in. Once that level (cp is achieved, learning on m2 can begin. 

Improvement above level Cj on facilitates improvement on m^^ Once the 

mastery level on m^ (c^) is achieved, additional learning on does not 

facilitate learning on m^. This relationship is illustrated in Figure I. 

This type of figure Is called a r.iodule characteristic curve, or MCC. 

For the relationship shown in Figure 1 , the vertical axis is che 
proportion of examinees passing m2. The hdrizdrital axis is the examinee's 

status (level of achievement) on m^. As can be seen, the relation ts 

horizontal until the level of achievement bri m^ designated by Cj is reached. 

At that pbint a linear relationship between status on and performance on m2 

is depicted. When the m^ mastery level, C2, is reached, a horizontal relation 

is again present, indicating that farther improvement on mj does not aid 

performance on 1119. Of course, the relationship in the range frbm c^ tb C2 

need not be a linear one, and in reality examinees would be expected to fall 
in a scatter around the curve shown in Figure 1. 

The low end bf the curve shbwri in Figure 1 is not at zero^ nor is the top 
at one. It would be expected that sbme small portion of examinees might pass 
m2 even with very little learning bri m^. This would be due to chance or other 

factbrs, arid wbuld gerierally be a small proportion of the total number of 
examiriees. It would also be expected that some Portion of examinees who had 
mastered m^ wbuld fail m2 , simply because nf failure to master the material 

riot iricluded in m^^. 

An Ill ustratlo a^ 

In order to illustrate the processes described above, simulation data 
were generated according to the following process. Item parameters for the 
three-parameter logistic (3Pt) model (Birnbaum, 1968) were selected for two 
thirty item modules. Examinee achievement levels were randomly selected 
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from a_ridrmal distributibri with a mean of zeirb and a standard deviation of 
1.5. The value was set equal to B (acHievemerit level) « -1.0^ arid was 

set eqiial to B= 0.5. Mastery of was arbitrarily defined as severiteeri 

correct but of thirty items; 

For each examinee, an m^, achievement level was selected as follows; If 

the examinee's m^^ achievement level ( 6p was less than Cp the examinee's 

achievement level (6^) was randomly selected from a normal distribution with a 

mean of -1.0, and a standard deviation of 0.5. If 0, > c,, but 9, < Co, 

II 1 z ^ 

was randbmly selected from a nbrmal distributibri with a mean of and a 

snandard deviation of 0.5 o If 9^ > c^, 9^ was randomly selected frbm a nbrmal 

distribution with a mean of 0.5 arid a staridard deviation of 0.5. Table I 
presents a summary of the relationship between 6^ arid 9^. Usirtfi; 9^ arid the 

mbdule 2 item parametcirs, refeporise data were ^erierated for mbdule 2 accbrdiri^ 
to the 3PL model for 1000 examinees. 



Figure 2 shows a plot of 8^ by 9^ for the 1000 simulated examinees; As 

can be seen, below -1.0 on the 6^ scale there is a correlation of about zero 

between 9^ and Between 9^ = -t.O and 0^ = 0,5 there is a positive 

correlation between 9^ and . Above 9^ = 0-5, there is again no correlation 

between 9^ and 9^. 

Figure 3 shows an empirical MCC fbr the generated data. The empirical 
MCC was computed by grouping examinees iritb iritervals of the ability scale bri 
the basis of 6 . For each interval, the prbpbrtibri of examinees in that 



interval who passed m2 was computed and plotted against the interval 

midpoint. As can be seen^ the plotted values form a rough approximation to 
the curve shox^m in Figure i. 



The Procedure 

The Mbdel 

The prbcedure prbpbsed for use with sequential units of instruction is 
based on the notion of the MCC. An MCC describes the probability of passing a 
unit bf instruction (module) conditional on latent ability (achievement level) 
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on the prerequisite module. The fdrni of the MCC propoiSGd in this p?3per ts the 
four-parameter Ib.cristic moHel, which is ^iven by 



where 0 , iJ5 the latent abilitv of examinee k bh rhbdiile i (the prerequisite 
ik 

module), Pj ( ^^j^) is the prbbahilifiy bf passing'; module j c^iven ability 6^^, 

is a discriminatibn parameter associated with module i, is a difficulty 

parameter associated with module j, Cj is a lower asymptote parameter for 

module -j , e^ is an upper asymptote parameter for module j, D = and EXP(x) 

= e^. The Cj term is used to account for the nonzero probability of passing 

module j for examinees with \?ery low ability on module i, and the ej term 

accounts for the nonur*ity probability of passing module :i fbr examinees bf 
very high ability on module i# 

Figure A shows a 4PL MCC. The a-parameter is related tb the slbpe bf the 
MCC at the point of inflection, while the b-parameter serves to Ibcate the 
point of inflection on the ability scale. 

Using the Model 

Interpreting the Parameters . Using the APL mbdel in conjunction with 
sequential units bf ihstructiori involves estimating ^nd interpreting the 
parameters bf the model. The slope of the MCC, as indicated by the a- 
Parameter, represents the strength of the sequential relationship. A steep 
slope indicates that small increases in achievement on the prerequisite module 
yield large increases in performance on the subsequent module ._ This wbuld be 
indicative of a strong sequential relationship. A relatively flat MCC 
indicates that even large increases in achievement on the first mbduledb hot 
yield substantial improvement in performance on the second module. This would 
be indicative of a weak sequential relationship. Thus, the a-parametier serves 
as an indicant of the strength of the sequential relationship. 

The b-parameter helps tb indicate what level of performance is required 
on the first module to attain a given lev'^l of perf brmahce bh the second 
module. If the b-parameter for the MCC shown in Figure 4 were increased, the 
curve V7buld be shifted to the right. If this were the case, a greater level 
of ability would be required on module 1 tb attain the same level of 
perfbrmrnce bh module 2 as was the case before the curve was shifted. Thusi 
the b-parameter Ibcates the mbdiile bh the achievement scale. 
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The c-parameter is a ' psendo-f^ue*5sin^ * parameter. tt_represerit:s the 
probability of passing nociiile 2 even when little or none of the material of 
module I has been masteriid. A larj^e value fpr c. indicates that much of the 
material of nodule 2 can be learrird without knowledcte of the material in 
mcjdule 1, Thus, the c-parameter is ah _indicant of the da^ree to which all of 
nodule 2 actaaiiy requires kriowIedTe of nodule 1 maferial. 

The e-paraineter is a reflection of the fact than mb-iule 2 contains 

instruction and material beyond those in nodule U "^erfecL mastery of module 

1 does not guarantee mastery of nodule 2. That is, module 1 is necessary but 
not sufficient for module 2. The p;reater the value of e , the more that nodule 

2 requires knowledge beyond what is required for iiiaster> of module I. 

Sattinc^ a Pass /Fail Score . The ^oai of setting a pass/fail cut score for 
nodule I is to minimize the number of examinees failini^ module _ 2 and to _ 
minimize the number of examinees ^^ho could have passed module 2 but are held 
hick. If these two types of errors are considered equally serious, then the 
most obvious procedure for setting a cut score for nodule 1 is to determine 
the level of ability on module 1 for which the predicted orbbability of 
success on nodule 2 is 0.5. Setting equation 1 equal to 0.5 and solvinf^ 
for B yields 



where 9 is the pass/fail cut score for module t, in(x) is the log to the base 
c 

e of X, and the other terms are as previously defined. 

Once estimates of the MGG parameters and examinee ability parameters have 

_ _ A _ _ 

been obtained, 6 is calculated from (2). Examinees for v7hom 9 (estimated 
c 

achievement) > 9 considered masters of module 1 arid are routed to module 2. 
c 

Examinees with 9 < 9- are considered ribnmasters arid are not allowed to proceed 

c 

to module 2. 



The ptocedare for estimating the item parameters of the 4PL model selected 
for this research is based on a maximum likelihood estimation Lechnique. An 
iterative procedure based on the Newton-Raphson approach to solving 
simultaneous nonlinear equations is employed. 




JEarameter Estimation 
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Cr 1 teri bh Fuhct ion 

The estimatibri procediire is Uesi.f^ned to maximize the criterion function 
giv^en by 



N a: t-u: 
L = n P. 0: ^ , (3) 

3 = t ^ ^ 



where L is the likelihood of the string of observed butcbmes (passes arid 
failures) for a module, N is the number of examiriees, u^: is the module oiitcdme 

(zerb for fail, one for pass) for examinee j, and is l-P^ • P^j is j^tven by 

(1). In practice, (3) is maximized by minimizing the nenativ? of the 
logarithm to the base e (natural lo^^arithm) of t . That is , L is mini mi zed 
where 



= - lo^r^(L) 



(4) 



Estimation Procedure 

The Newton-Raphson procedure employed requires the first and second 
partial derivatives of (4), taken with respect tb the item parameters. If 
is a column vector bf first derivatives, arid _r* is the matrix of second 
derivatives i then fbr any set bf provisional item parameter estimates, updated 
estimates are bbtained usirijt the fblldwirig formula: 



:i+t 



= f 



Id' 



(5) 



V7here f^ is the vector bf item parameter estimates after iteration i, arid f_ 

is the vector of item parameter estimates after iteraticri i -h 1 • The first 
and second derivatives bf (4) are ^iven in the Appendix. In a ^iven 
iteration^ these derivatives are evaluated usiri^ the estimates from the 
previous iteratibri. 

Orie prbblem which is encountered in a procedure iikr^ this occurs when the 

matrix bf second derivatives, given by is not positive definite. The 
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Newtbri-Raphsdri prbcedura jTuararitees convergence only when is niways 
positive definite. Wliian a model such as the 4PL model is used, the matrix of 
second derivatives, evaluated at the provisional item parameter estimates ^ . 
very often is riot positive definite; therefore, it is necessary to check _f_" 
for positive def iriiteness • Tf it is not positive definite, it should be 
forced to be positive definite, A namber of procedures for doing this have 
been proposed; 

Work is currently underway on a program implementing the above estimation 
procedure. At this point research is underway to determine the optiir.al 
procedure for forcing the matrix of second derivatives to be positive 
definite. It io hoped that a working version of the program will be available 
shortly . 

Examp le 

In order to illustrate the bperatibri of the estimation procedure iust 
described, a preliminary version of the estimation program was applied to 

the simiilatibri data generated in the previous section of this paper and for 
which the empirical MCC is shown in Figure 3. The true m, achievement levels 



were used as input to the estimation program; 

Table 2 shows the item parameter estimates which resulted from_the 
application of the 4Pt estimation ' program to the simulation data. Figure '5 
shows the empirical MCC shown in Figure 3, with an overlay of the theoretical 
MCC computed using the item parameter estimates shown in Table 2. As can be 
seen, the theoretical curve shown in Figure 5 provides a reasonable 
description of the observed data. 

Evidence for the Validity of the Model 



Meth od 

For the purposes of acquiring evidence to either support or discredit the 
4PL model and the MCC concept ^ real respbrise data were collected for a two- 
part arithmetic test. It was hypbthesized that the two parts of the test were 
siich that the skills required fbr perfbrmarice on the first part ^otjl^^e 
prerequisite tb perfbrmarice bri the secorid part. Using these two parts as 
nbdiiles, empirical MCCs were plbtted for various pass/fail cutoffs on the 
secbrid module. These plots were then examined as evidence of the usefulness 
of the 4PL model for use with tnese data. Details of the process follow. 

Data . The test used for these analyses was the Numerical Skills subtest 
of the Career Placement Program (CPP) test (the American College Testing 
Program, 1983). tho first part of the test, module 1, is comprised of 
nineteen four-choice multiple-choice arithmetic combutatibh prbhlems, while 
the second part J module 2y is comprised of thirteen fbur-chbice multiple- 
choice word problems that require arithmetic cbmputatiori skills and ptoblem- 
sblvirig skills. Response data fbr these items were cbllected for 3768 cases 
from the 1983 ribrming admiriistratibris of the test. 



Sirice there is rib already determined pass/fail cutoffs for the CPP 
subtests, this arialyses performed in this stage of the research were repeated 
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for a nunber of different cutoffs for moduie 2, so as to avoid any 
capitalization on chance fron the cutoff selectioni Usin^r a g;iven pass/fail 
catoff for module 2; each examinee was assi?;nGci a score of 0 (fall) or 1 
(pass ) depending on whe ther the examinee 's raw score on module 2 exceeded t he 
cutoff for module 2i These 0, i data, alonp; with examinees* achievernent level 
estimates from module I, were the input for these analyses. 

Ability Estimation ^ The achievement level esr.imates on nodule I for the 
examinees were obtained through the application of the 3PL model to the 
examinees' response data for module 1. The LdGIST estimation proji^ram 
(V/in^ersky , Barton, and Lord, 1982) was used to estimate the paramecers of the 
3PL model. 

Plotting MCCs , The initial step in these analyses was the division of the 
achievement scale into a number of narrow intervals (0.1 width). Examinees 
were then sorted iritb these intervals bri the basis of their module 1 
achievement level estimates. For a divert module 2 pass/fail cutoff, the 
proportion of examinees within each interval passing module 2 was computed. 
For each module 2 cutoff, the proportions passing module 2 were plotted 
aj^ainst the interval midpoints, thus forming an empirically derived MGC. 
Adiacent intervals were collapsed to assure an interval sample size of at 
least ten. These MCCs were exc.mined to assess the reasonableness of the 4PL 
model for describint^ the form of the resulting curve. 

Results 

Figures 6 through 12 show the empirical MCCs obtained for the CPP data 
for pass/fail cutoffs on module 2 of three through nine correct out of the 
thirteen items, respectively. Table 3 shows tlie obtained proportions passing 
plotted in Figures 6 through 12. Table-. 3 also shows the numbers of examinees 
in the different intervals. 

As can be seen from these figures, the relationship betvjeen module I 
ability and module 2 performance does appear to be at least a monotonically 
increasing one. Also, for several of the plots, there appears to be a non- 
unity upper asymptote. It is, however, diffirulc to discern a lower asymptote 
in these plots. Of course, a lower asymp'-ote of zero is a special case of the 
4PL modeli It may eventually be fruitful to drop the lower asymptote, but as 
yet there is little evidence to support such a steps 

There are a couple of" interesting trends evident in Figures 6 through 
12* As the pass/fail cutoff score on module 2 increases, of course, fewer 
examinees of low achievement level on module 1 yass nodult!_2. If the material 
in module 2 requires the knowledge of_ module I material^ clearly requiring 
more module 2 material for passing will require more module 1 material. 

As the module 2 pass/ fail cutoff increases , the upper asymptote of the 
MCC decreases (the e term increases in value). This is ari indication that 
complete knowledge of module I is not sufficient for guaranteed success on 
module 2. Another way of saying this is that word oroblems require more 
knowledge than; simply mastering arithmetic operations. 



The patterns evident in these figures sag^rest that, if module 2 were 
still easier to pass than was the case with the pass/fail score of 3, there 
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would he a nonzero lower asymptote to "the fiCC. Unf or tiiriately , for this 
partlciilar test lower cutoffs yielded an almost fiat MCC near unity. Almost 
all examinees f^ot at least two items correct on module 2^ re^^ardless of their 
module 1 abiiity. 

Summary and Con clus i oris 

While tliis research project is still incomplete, it has yielded 
encouraging results. A theory relating performance on sequeritiially arranged 
units of instruction was derived^ and a model for describLhg that relationship 
was formulated. Procedures for using the model to evaluate sequential 
relationships and for making routing decisions V7ere described . A procedure 
for estimating the parameters of the model was outlined, and data supporting 
the validity of the motlel were presented. All things considered, the model 
and procedures described appear to be useful dries, arid they appear to merit 
continued research efforts directed toward their development. 
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Apperirlix 

Derivatives dt* thia Nia^ative of the Natural 
Logarithm of the Criterion Fuhctibri 



The negative of the natural logarithm of the criterion function, denoted 

_ ^ _ _ _ _ _ — 

bv L , was (i;iveri by (4). The vector of first derivatives with respect to the 

ttpn par?imeters , denoted by f ' , is Riven by 



r 



f' = 



9a ^ 



3b 



^ L 

8c _ 



(P. - c)(u: - P. ) 



D Z (9. - b) ~ ' '^""r 

_j = l J P. Q. [1 + FXP (X. ) J 

N (P. - c)(u. - P.) 
'^^ P. nil + EXP (X^. ) I 



u. - P 

1 



. . P. 0. t 1 + EXP (X. ) i 

^^ (u. - p. ) EXP (X. ) 

- y — ^ .1 .1 

.-^1 Pj O.Ll + EXP (X.)J 
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where P. = Pj ( 5. ^ ) , Q-j = I " P-| . = ba( 6. - b), arid t he remaining teims are 
as defined for fl). the matrix of seconfi derivatives, denoted by _f_" , is friven 
hy 



£" = 



3?r^ t 



8a 3b L 



32 



32 



9a 8c L 



32 



3b 3c L 



-a2_ 



3a 3e L 



32 



SbSe L 



32 



Ic^ L 3c3e L 



a2 



The matrix is symmetric. The individual terms in the matrix are given by: 



92 , 

^ L 



, ^ - Ou.c - P.2q. - p_e EXP(x.3(u - p ) 
-d2 E (9. - b)2 (P. - c) 1 ..^ J , , .1 ,^ .i -i 



j = l 



P^^ Qj'' [1 + EXP (x,)j^ 



Sb' 



D2a2 



N 

r (P 
j = i ^ 



- c) 



p^2q. 



Q^cuj + P,e EXP(x.)(u^ 



p7^ 

J 



^. [1 + EXP (x.)J'^ 



ii 



N 

3 = 1 



- 2P. 



^ 



J 



P.^ 0,-^[l + EXP (x ) |^ 



ii 



N p,2 
* = E EXP(2x.) 



- 2P u. 4- u 

J .1 



^ ..rvzx ; ; EXP(X. ' 



13 



13 



D E (P 

.1 = 1 ■ 



- c) 



P:^ O:^ I + EXP (x. ) 1^ 



P:) 

-1 



a: - P. 

.1 .1 



P. Qj [1 + EXP (Xj)jr ' 



5 _ P;Q.u; - Q:cu. + P.e EXP (x,) (u - P ) 
D Z (9. - b) ^^^^ 3 ? 1 ^ 1 i y 

:i = l ^ 



P.^ Q.^ 11 4- EXP (x.) 1^ 



_ P,^ - p^2c - p 2u - d cu + P cu 

-D E (9. - b) EXP(x-) -^7 > ] •'' ^77 ^ 

J Vi P,^ 0.^ [i + EXP (x.)J^ 



.1 = 1 



.1 



-Da 



N 

T. 

d = i 



Q;u. - 0 cu 

- J -1 1 



+ P:e EXP (x. )(a. - P ) 
■1 11 1 

P.^ 0.^ 1-1 + EXP (x.) 1^ 



Da 



N P.^ - P.^u. - P,2c -f P.cu. - Q.cu 

^""j^ P:^ Q.^ 1 -f EXP (x.)J2 ; 



and 



N P.^ + P, u. + Q,u. ' 
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Table i 



Summary of Relationship Between (3^ and 6^ 



> N(^i.n, 0.5) 

^2 ^ ^7-^ M(n.5, 0.5) 



Table 2 

Item Parameter Estimates 
for Simulated 4PL Data 



Parameter Estimate 



? 1.175 

b -0.160 

c • 0.021 

e 0.076 
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Table 3 



interval 



Samp'' Sizes and Prbpbrtibris. Passing 
for Hach Achievement Interval on rij 



SaniPle Cutoff on M2 



Size 



8 



I ^ 0.000 0.000 0.000 o.oon n.oon o.onn 



0.000 0.000 o.poo 



2 ib 0.100 n.Doo o.ono n.ono 

I \l 0-333 0.133 0.057 O.OOO 0.000 0.000 O.'oOD 

X In ^'^"^ -'--^ 0-000 0.000 O.DUO 

^ 20 0.250 0.200 0.100 0.000 0.000 0.000 0.000 

5 16 0.250 0.125 0.000 O.OOO 0.000 0.000 0.000 

/ 13 0.231 0.077 0.000 O.OOO 0.000 0.000 0.000 

? 26 0.423 0.192 0.038 O.OOO 0.000 0.000 0.000 

.1 34 0.382 0.265 0.088 0.059 0.029 0.000 0.000 

!? 29 0.414 0.207 0.103 0.069 0.034 0.000 0.000 

0.000 
0.000 
0.O23 0.000 
O.OOO 
0.000 0.000 
0.015 
0.029 0.029 
0.023 



}J 34 0.294 0.118 0.000 0.000 0.000 0.000 

\l 32 0.375 0.250 0.094 0.063 O.OIt 0.031 

^3 0.558 0.395 0.186 0.047 0.047 0.023 

]-* §0 0.400 0.200 0.150 0.067 0.000 O.OOO 

\l 59 0.525 0.305 0.119 0.051 0.017 0.000 
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Figure 2 

Relatidhship between Achievement teveis 
on Modules 1 and 2 




18 

o 

ERIC 



Figure 3 

Empirical MCC for Generated Data 
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Figure 4 
A 4PL MCC 
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Figure 6 



Empirical MCC for CPP Data 
Cutoff = 3 
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Figure 7 

Empirical MGC for GPP Data 
Gutoff ^ 4 
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Figure 8 
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Empirical MCC for CPP Data 
Cutoff = 5 
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Figure 9 

Empirical MCC for CPP Data 
Cutoff = 6 
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Figure iO 



Empirtcai MCC for GPP Data 
Catoff = 7 
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Empirical >tCC_for CP^^ Data 
Cutoff = 8 
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Figure i2 



Enpiricai MeC for CPP Data 
Catoff = 9 
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A tatent trait Model for Use with Sequentially 
Arranged Bnits of Instruction 



Abstract 



A theory relating performance on sequentially arranged units of 
instruction was derived, and an item response theory model for describing that 
relationship was formulated. .Procedures for using the model to evaluate 
sequential relationships and for making routing decisions were described. A 
procedure for estimating the. parameters of the model was outlined^ and data, 
supporting the validity of the model were presented. Overall, the model arid 
procedures appeared to be useful ones^ arid they appeared to merit cbritiriued 
research efforts directed toward their further developmerit. 
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